Skip to content

HDFS-17655. Cannot run HDFS balancer with BlockPlacementPolicyWithNodeGroup#7141

Merged
Hexiaoqiao merged 1 commit intoapache:trunkfrom
eubnara:HDFS-17655
Dec 26, 2024
Merged

HDFS-17655. Cannot run HDFS balancer with BlockPlacementPolicyWithNodeGroup#7141
Hexiaoqiao merged 1 commit intoapache:trunkfrom
eubnara:HDFS-17655

Conversation

@eubnara
Copy link
Contributor

@eubnara eubnara commented Oct 31, 2024

Description of PR

When I use BlockPlacementPolicyWithNodeGroup, I cannot run balancer.
It is a side effect of HDFS-15363.

How was this patch tested?

Manual test.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

@hadoop-yetus
Copy link

hadoop-yetus commented Nov 1, 2024

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 43m 47s trunk passed
+1 💚 compile 1m 16s trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 compile 1m 9s trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 checkstyle 1m 1s trunk passed
+1 💚 mvnsite 1m 17s trunk passed
+1 💚 javadoc 1m 3s trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javadoc 1m 40s trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 spotbugs 3m 14s trunk passed
+1 💚 shadedclient 41m 30s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 20s the patch passed
+1 💚 compile 1m 21s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javac 1m 21s the patch passed
+1 💚 compile 1m 16s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 javac 1m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 0s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)
+1 💚 mvnsite 1m 23s the patch passed
+1 💚 javadoc 1m 1s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javadoc 1m 42s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 spotbugs 3m 39s the patch passed
+1 💚 shadedclient 44m 12s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 221m 13s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 43s The patch does not generate ASF License warnings.
372m 59s
Reason Tests
Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/1/artifact/out/Dockerfile
GITHUB PR #7141
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux b16d8c6d3140 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cf7d519
Default Java Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/1/testReport/
Max. process+thread count 4005 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@Hexiaoqiao
Copy link
Contributor

Hexiaoqiao commented Dec 8, 2024

Try to trigger CI manually, let's wait to what it will say.

@hadoop-yetus
Copy link

hadoop-yetus commented Dec 8, 2024

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 32m 14s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 43m 38s trunk passed
+1 💚 compile 1m 11s trunk passed
+1 💚 checkstyle 1m 10s trunk passed
+1 💚 mvnsite 1m 19s trunk passed
+1 💚 javadoc 1m 48s trunk passed
+1 💚 spotbugs 3m 14s trunk passed
+1 💚 shadedclient 34m 10s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 6s the patch passed
+1 💚 compile 1m 2s the patch passed
+1 💚 javac 1m 2s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 58s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)
+1 💚 mvnsite 1m 7s the patch passed
+1 💚 javadoc 1m 31s the patch passed
+1 💚 spotbugs 3m 11s the patch passed
+1 💚 shadedclient 34m 12s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 222m 24s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 46s The patch does not generate ASF License warnings.
382m 30s
Reason Tests
Failed junit tests hadoop.hdfs.TestDecommission
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/2/artifact/out/Dockerfile
GITHUB PR #7141
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 95a19f8bcbbc 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cf7d519
Default Java Red Hat, Inc.-1.8.0_412-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/2/testReport/
Max. process+thread count 3820 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/2/console
versions git=2.9.5 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@Hexiaoqiao
Copy link
Contributor

Hexiaoqiao commented Dec 9, 2024

Hi @eubnara do you mind to check if the failed unit test relate to this PR. Thanks.

@eubnara
Copy link
Contributor Author

eubnara commented Dec 11, 2024

I checked it but I think the error in TestBalancerRPCDelay, TestReadStripedFileWithDecodingDeletedData is not related to this PR.
I tested it on my laptop with the same code but the two tests are passed.

image

image

image

@hadoop-yetus
Copy link

hadoop-yetus commented Dec 12, 2024

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 43s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 38m 2s trunk passed
+1 💚 compile 1m 6s trunk passed
+1 💚 checkstyle 1m 6s trunk passed
+1 💚 mvnsite 1m 12s trunk passed
+1 💚 javadoc 1m 42s trunk passed
+1 💚 spotbugs 2m 59s trunk passed
+1 💚 shadedclient 37m 58s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 7s the patch passed
+1 💚 compile 1m 1s the patch passed
+1 💚 javac 1m 1s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 56s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)
+1 💚 mvnsite 1m 9s the patch passed
+1 💚 javadoc 1m 33s the patch passed
+1 💚 spotbugs 3m 13s the patch passed
+1 💚 shadedclient 34m 1s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 222m 32s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 47s The patch does not generate ASF License warnings.
348m 52s
Reason Tests
Failed junit tests hadoop.hdfs.TestDecommission
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/3/artifact/out/Dockerfile
GITHUB PR #7141
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux e77895c5c536 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cf7d519
Default Java Red Hat, Inc.-1.8.0_412-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/3/testReport/
Max. process+thread count 4060 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/3/console
versions git=2.9.5 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@haiyang1987
Copy link
Contributor

haiyang1987 commented Dec 15, 2024

Thanks @eubnara work, please fix checkstyle.

@hadoop-yetus
Copy link

hadoop-yetus commented Dec 20, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 11m 47s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 35m 16s trunk passed
+1 💚 compile 1m 19s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 1m 10s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 1m 10s trunk passed
+1 💚 mvnsite 1m 19s trunk passed
+1 💚 javadoc 1m 9s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 39s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 3m 8s trunk passed
+1 💚 shadedclient 34m 47s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 8s the patch passed
+1 💚 compile 1m 10s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 1m 10s the patch passed
+1 💚 compile 1m 2s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 1m 2s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 59s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)
+1 💚 mvnsite 1m 9s the patch passed
+1 💚 javadoc 0m 57s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 30s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 3m 2s the patch passed
+1 💚 shadedclient 34m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 221m 36s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 46s The patch does not generate ASF License warnings.
358m 51s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/4/artifact/out/Dockerfile
GITHUB PR #7141
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 7fe9be121523 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 155fff0
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/4/testReport/
Max. process+thread count 3628 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao
Hexiaoqiao approved these changes Dec 21, 2024
View reviewed changes
Copy link
Contributor

@Hexiaoqiao Hexiaoqiao left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGMT. +1 from my side. Will commit if no more other comment while wait two work days.

@Hexiaoqiao Hexiaoqiao merged commit d5b836b into apache:trunk Dec 26, 2024
2 checks passed
@Hexiaoqiao
Copy link
Contributor

Hexiaoqiao commented Dec 26, 2024

Committed to trunk. Thanks @eubnara for your works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Hexiaoqiao Hexiaoqiao Hexiaoqiao approved these changes

Assignees

No one assigned

Labels

HDFS trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@eubnara @hadoop-yetus @Hexiaoqiao @haiyang1987